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ABSTRACT 

The purpose of this study was to research the effect of the rubric on the level of learners’ cognitive engagement and to 
explore the effectiveness of Web-based concurrent discussions experienced by learners. The participants were 20 
undergraduate students of D Women’s University and they were divided into six groups for concurrent discussions in 
Web-based environment. Three groups of 10 participants were provided with rubric I to conduct self-evaluation and the 
remaining three groups of 10 participants were provided with rubric II to conduct self-evaluation and peer evaluation. 
After concurrent discussion in Web-based environment, it was found that the degree of learners’ cognitive engagement 
using rubric II showed higher-level cognitive engagement than the other groups using rubric I. In addition, with regard to 
the level of learners’ cognitive engagement, the groups using rubric II interacted more frequently with one another than 
the other groups using rubric I. 
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1. INTRODUCTION 

Web-based discussion is considered important in that it contributes to expanding opportunities of 
convergence through the text-based interaction and promoting critical thinking and problem solving skills, 
which are important elements to lead successful learning in STEAM education (Lim, 2005; Lim, Yeon, & 
Kim, 2011; Harasim, 1990). Besides, the contents of discussion in Web-based discussions can be modified 
because the communication is text-based (Lee & Yang, 2009) and introspective learners who were difficult to 
participate in face-to-face discussions will be more active (Koh, 2004). In addition, it has positive effects 
such as the possibility of many-to-many discussions (Harasim, 1990), reduction of the impact of non-verbal 
elements like appearance and voice (Weisskirch & Milburn, 2003), mitigation of opinion collisions by plain 
emotional expressions, and the record of the contents of the discussion (Kyungsun Hong, 1999). Despite 
these advantages, however, most of learners experience difficulty in Web-based discussions. Therefore, 
various studies regarding design strategies for Web-based learning environment to promote discussions and 
to develop support tools are required in order to obtain an intended level of learning performance effectively 
(Park, 1998; Lim, Yeon, & Kim, 2011; Harasim, 1990). 

One of strategies for overcoming the difficulties of Web-based discussions and eliciting the learning 
effect is to provide scaffoldings to support the discussion process. Among them, the rubric has been widely 
utilized as a strategy for leading discussions effectively and promoting cognitive engagement of a higher 
level such as critical thinking and problem solving skills. In addition, the evaluation area in the rubric is 
presented more precisely by describing the execution criteria in detail and it is possible for learners to 
enhance their critical thinking by evaluating objectively their and others’ opinions. In particular, it can 
promote spontaneous interaction and lead the discussion effectively by inducing the learners’ active 
participation in the discussion and providing opportunities for checking up the demonstration process with 
clear evaluation criteria (Toth, Slithers, & Lesgold. 2002). This study intended to improve the level of 
cognitive engagement of learners utilizing the rubric in the Web-based concurrent discussions by focusing on 
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the effect of these characteristics of the rubric. In addition, this study was to explore learners’ experiences in 
the progress of Web-based concurrent discussion. For this, the major research questions are as follows: 

First, what are the effects of the rubric type on the level of learners’ cognitive engagement in Web-based 
concurrent discussion? 

Second, how do the learners recognize the effectiveness of Web-based concurrent discussions using the 
rubric? 


2. METHODS 

2.1 Participants 

The participants were twenty undergraduate students in a college of education. There were female students 
who took the course titled ‘Methodology and Technology of Education.’ This is a required course for 
pre-service teachers’ training. Most of the students were juniors and had prior knowledge related to learning 
theories such as behaviorism, cognitivism, and constructivism. The students were randomly assigned to one 
of six groups. 

2.2 Procedures 

The independent variable of this study was the type of rubric and the dependent variables were the level of 
learners’ cognitive engagement and the effectiveness of Web-based concurrent discussion using the rubric. 
The research design of the study is shown in Table 1. 


Table 1. Research design 


Gl 

01 

XI 

02 

G2 

01 

X2 

02 


Gl: the group of self-evaluation 

G2: the group of self-evaluation and peer-evaluation 

XI: rubric I for self-evaluation 

X2: rubric II for self-evaluation and peer-evaluation 

Ol: pretest to measure the participants’ prior knowledge 

02: posttest to measure the effectiveness of the rubric 

To examine the effect of the rubric for Web-based concurrent discussion, this study was conducted 
through the procedure as in Table 2. 


Table 2. Research procedure 


Phase 

Procedures 

Research design 
Experimental tool development 
Group design 
Pretest 
Experiment 
Posttest 
Data analysis 

Design of the research model 
Development of test tools 
Placed in the experimental and control groups 
Check the homogeneity of the two groups 
Web-based concurrent discussion 
Measure the effectiveness of the rubric 
Analysis of the level of cognitive engagement and the effectiveness 


2.3 Research Tools 

2.3.1 Web-Based Concurrent Discussion Environment 

Web-based concurrent discussion environment was implemented using the cafe service of portal site Daunt in 
consideration of user-friendliness and accessibility for the teacher and learners (see Figure T). 
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Figure 1. Web-based concurrent discussion environment 


2.3.2 Pretests 


A pretest was used to measure the learners’ knowledge before the Web-based concurrent discussion, and the 
test consisted of ten multiple-choice questions. The questions were related to discussion topics such as 
behaviorism, cognitivism, and constructivism. The reliability coefficient of the pretest (Chronbach’s alpha) 
was .51 and identity verification results showed homogeneity between the two groups (F=.29, p=.60). 

2.3.3 Rubric 

The rubric has been provided to support self-evaluation and peer-evaluation. Based on the previous studies of 
Paul and Elder (2013), this study used the rubric to develop subjective questionnaires. The rubric used the 
same questions for both self-evaluation and peer-evaluation and a 5-point Likert scale was developed to 
measure five items of the rubric such as clarity, accuracy, relevance, logical order, and depth. 

2.3.4 Cognitive Engagement Level 

Analysis of the level of learners’ cognitive engagement in Web-based concurrent discussion used Lee’s 
(2005) cognitive engagement analysis system of online learning. It consisted of four items such as criticism, 
expansion, repetition, and unproductivity. In addition, the Discussion Analysis Tool (DAT) by Jeong (2005) 
was used in order to investigate the effect of the rubric on the flow of Web-based concurrent discussion. 

2.3.5 Effectiveness of Web-Based Concurrent Discussion 


The perceived effectiveness of Web-based concurrent discussion using the rubric was measured after the 
discussion through open-ended questions. The open-ended questions were about how the learners recognized 
the effectiveness of Web-based concurrent discussion and the rubric. 


2.4 Data Collection and Analysis 

The dependent variable analyzed in this study was the level of cognitive engagement and the independent 
variable was rubric type. Lor the comparative analysis of rubric types, '//-test was conducted by PSAW 
Statistics 18 and the significance level for statistical verification was set to .05. In addition, the DAT of Jeong 
(2005) was used to analyze interaction between the cognitive engagement level and the flow of discussion. 
Questionnaire analysis was conducted in order to explore the perceived effectiveness of Web-based 
concurrent discussion using the rubric (see Table 3). 


Table 3. Research procedure 



Data 

Data collection 

Data analysis 

The level of cognitive 

Discussion 

Conversations occurring 

X 2 -test 

engagement 


in the progress of 
discussion 

DAT analysis 

Perceived effectiveness 

Open-ended questions 

Survey after discussion 

Questionnaire analysis 
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